The present invention relates to log storage management in a data processing system with a key-pointing operation, and particularly to a mechanism for allocating storage for key-pointing.
In certain data processing operations it is useful to maintain a log related to ongoing data processing activity. For example, in the transaction processing field, it is common to maintain a log of ongoing transactions involving a particular data processing apparatus so that if the data processing apparatus should fail for some reason (e.g., power failure) the log can be used to recover the data processing apparatus back to the state it was in, with respect to processing the transactions, before the failure. This is a very important consideration because the business world depends on the processing of transactions to be tolerant of system failures in order to maintain the integrity of mission critical data which is involved in the transactions.
Such logs are typically maintained on a direct access non-volatile storage device (such as a hard disk drive) so that data will not be lost in the case of power failure. As the data processing apparatus processes ongoing transactions, log records containing information pertaining to the ongoing transactions are written sequentially into one or more pre-allocated log files (hereinafter called xe2x80x9cextentxe2x80x9d files). These log records would thus be retrieved from the extent files to reconstruct the transaction processing environment within the data processing apparatus in the event of a failure to enable a subsequent recovery from such a failure.
The size of an extent file can be configured by the operator and log records are written to an extent file up to the configured size, at which point a new extent file is created in order to store more log records. It is often the case that an extent file is very large as compared to a log record, and thus many log records are stored in a single extent file. Special xe2x80x9clinkxe2x80x9d records are used to link one extent file to the next. A spare extent file known as a xe2x80x9ccushionxe2x80x9d file is also pre-allocated. This cushion file can be used as an extent file instead of creating a new extent file in the event that the log runs short on disk storage.
To prevent the storage requirements of the extent files continually increasing as more log records are written, an operation known as xe2x80x9ckey-pointingxe2x80x9d is used. This operation takes advantage of the fact that stored log records become redundant. For example, in transaction processing, when a transaction has completed there is no longer a need for the data processing apparatus to store log records for that transaction and there is no longer a need for that transaction to be taken into account in the event of recovering from a failure. As a result, the log records pertaining to that transaction are redundant. If the data contained in a log is DW, it will consist of the data associated with completed transactions (DC) and the data associated with active transactions (DA). It is obvious then that the size of DA will be less than or equal to DW. The key-pointing operation involves rewriting all of the active data (DA) to the log as follows:
1. write a key-point start record to the log.
2. rewrite all the active data (DA) to log.
3. write a key-point end record to the log.
4. release or reassign storage the storage that previously contained DW.
This leaves the log with only the records relating to active transactions.
One example of a key-pointing system is shown in GB 2344196 in which key-points are taken only when an extent becomes full, in order to reduce unnecessary processing. However, in this and other key-point processing systems that handle large numbers of concurrent transactions, it is possible for the size of DA to become very large. As a result it is possible for the storage on which the extent files are maintained to run out such that no new log records can be written. If this happens the processing of a transaction can no longer tolerate a system failure and the transaction process can no longer function. The cushion file provides some measure of protection against this happening, however, the cushion file does not provide full protection as the amount of extra space it provides is arbitrary and it too can be used up.
The present invention addresses the aforementioned problem of key-pointing, namely of storage becoming exhausted such that new records can no longer be written to the log.
According to a first aspect, the present invention provides a data processing method for use on a data processing system having a direct access non-volatile memory storage device for storing log records thereon in one or more log files, the method comprising the steps of: writing log records to said log files when requested; performing a key-pointing operation on the written log records when a predetermined maximum cumulative size of log records, up to the first key-pointing operation or since the last key-pointing operation, is reached; calculating the amount of storage required to continue writing log records to the completion of a future key-pointing operation; and allocating said required amount of storage.
According to a second aspect, the present invention provides a computer program product for running on a data processing system having a direct access non-volatile memory storage device for storing log records thereon in one or more log files, the program product, in operation, carrying out the steps of: writing log records to said log files when requested; performing a key-pointing operation on the written log records when a predetermined maximum cumulative size of log records, up to the first key-pointing operation or since the last key-pointing operation, is reached; calculating the amount of storage required to continue writing log records to the completion of a future key-pointing operation; and allocating said required amount of storage.
According to a third aspect, the present invention provides a data processing apparatus comprising: a direct access non-volatile memory storage device for storing log records thereon in one or more log files; a writing means for writing log records to said log files; a key-pointing means for performing a key-pointing operation on the written log records when a predetermined maximum cumulative size of log records, up to the first key-pointing operation or since the last key-pointing operation, is reached; calculation means for calculating the required amount of storage to continue writing log records to completion of a future key-point; and allocation means for allocating said required amount of storage.
By providing a data processing system with the capability to predict log storage requirements and because the amount of data written to the log before a key-point is fixed, the maximum size of that key-point can be predicted, and so the storage requirement to completion of that key-point can also be predicted. This can be extrapolated to forecast the maximum log storage requirement to any future key-point operation. As a result storage can be allocated to guarantee log operation to a chosen point in the future.
Preferably the allocating step of the data processing method reuses some or all of the storage that becomes available as a result of a key-pointing operation. This provides an additional advantage because it provides the data processing system with the capability to reuse allocated storage after the data it contains becomes redundant. This can reduce the total log storage requirement and is very useful in a data processing system that has a log storage requirement that does not exceed a particular value. Once the storage has been obtained to meet this requirement it is retained.
In addition, in a system that reuses storage and stores log records in more than one log file it can be an advantage to start a key-point in a new log file. This can be beneficial when the granularity of storage reuse is per file. In this situation a file is made available for reuse a soon as the log records that it contains become redundant, reducing the overall storage requirements of the log.
According to another preferred feature, the system can operate in a reduced mode in response to a predetermined condition, wherein during said reduced mode only log records that will be redundant in the next key-pointing operation are written to the log. This provides a significant advantage to a system as it provides the capability to reduce the size of the active data and therefore reduce log storage requirements, when considered necessary.
The system is further improved if the allocating step is carried out immediately after a key-pointing operation is performed and the predetermined condition for operating in a reduced mode of operation is this allocation failing because of the unavailability of sufficient storage. This ensures that the system starts operating in the reduced mode as soon as a storage restriction is noticed and, if the system also includes the capability to reuse allocated storage after the data it contains becomes redundant, the system is guaranteed to have log space available whilst operating in reduced mode. In addition, if the log records written by the system to reduce the size of the active data are smaller than the log records of the active data, there is guaranteed to be enough space available to reduce the log storage requirements to a level such that the reduced mode can be ended.
Once reduced mode is entered, detecting, as soon as possible, that the storage restriction is no longer present, either because extra storage becomes available or because the log storage requirement has been reduced to a workable level, is an advantage. This can be achieved when the reduced mode of operation is active by performing a key-pointing operation when a record is written to the log that makes some previously active data redundant. If the system also attempts storage allocation after each key-point operation the lifting of the storage restriction will be detected as soon as possible.
If the data processing method is a method of transaction processing whereby the log records represent transactional data that can be used to reconstruct a transactional environment that previously existed, it is preferable that the method further comprises the steps of: rejecting a request to start a new transaction when operating in said reduced mode; and rejecting a request to involve when operating in said reduced mode;
This enables the system with a transaction log to take advantage of the restriction means by only writing records to the log that relate to the completion of transactions when the reduced mode is active.